Papers with mixture of experts
CLER: Cross-task Learning with Expert Representation to Generalize Reading and Understanding (D19-58)
Copied to clipboard
| Challenge: | In-domain datasets are used to train and validate our model, and other out-of-domain data are used for validation. |
| Approach: | They propose a model which uses cross-task learning with expert representation for the generalization of reading and understanding. |
| Outcome: | The proposed model achieved an average F1 score of 66.1 % in the out-of-domain setting, which is a 4.3 percentage point improvement over the official BERT baseline model. |
Two Heads are Better than One: Nested PoE for Robust Defense Against Multi-Backdoors (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing defense mechanisms assume that only one type of trigger is adopted by the attacker, while defending against multiple simultaneous and independent trigger types necessitates general defense frameworks. |
| Approach: | They propose a framework that uses a mixture of experts as a trigger-only ensemble to defend against multiple trigger types. |
| Outcome: | The proposed framework defends against multiple trigger types in a single ensemble and in combination of models. |
ASEM: Enhancing Empathy in Chatbot through Attention-based Sentiment and Emotion Modeling (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing models lack feature representations that capture the deep semantics of language and sensitivity to minor input variations, resulting in significant changes in the generated text. |
| Approach: | They propose an end-to-end model architecture called ASEM that performs emotion analysis on top of sentiment analysis for open-domain chatbots. |
| Outcome: | The proposed model outperforms existing models for generating empathetic embeddings, providing e-mpathetic and diverse responses. |
A Survey on Dynamic Neural Networks for Natural Language Processing (2023.findings-eacl)
Copied to clipboard
| Challenge: | Dynamic neural networks can scale up pretrainable models with sub-linear increases in computation and time. |
| Approach: | They summarize the progress of three types of dynamic neural networks in NLP . skimming, mixtures of experts, and early exit are among the most popular . |
| Outcome: | The proposed models can scale up with sub-linear increases in computation and time . skimming, mixture of experts, and early exit are the most popular approaches . |
Manifold-Preserving Transformers are Effective for Short-Long Range Encoding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Multi-head self-attention-based Transformers have shown promise in different learning tasks . but encoders of Transformers and their variants fail to preserve layer-wise contextual information . |
| Approach: | They propose an encoder model that guarantees a theoretical bound for layer-wise distance preservation between a pair of tokens. |
| Outcome: | The proposed model preserves equivalence between tokens and performs better than Transformers. |
QaDialMoE: Question-answering Dialogue based Fact Verification with Mixture of Experts (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing research on fact verification focuses on news, tables and Wikipedia passages. |
| Approach: | They propose a question-answering dialogue based fact verification with mixture of experts that exploits questions and evidence effectively in the verification process. |
| Outcome: | The proposed approach outperforms previous approaches on three benchmark datasets and achieves state-of-the-art results. |
Lifting the Curse of Capacity Gap in Distilling Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Existing studies have shown that pretrained language models require a tremendous amount of inference compute to perform. |
| Approach: | They propose to compress pretrained language models to small ones with a teacher-student paradigm to fill the capacity gap. |
| Outcome: | The proposed model achieves state-of-the-art performance at small FLOPs compared with competitive baselines. |
Mixture Content Selection for Diverse Sequence Generation (D19-1)
Copied to clipboard
| Challenge: | Generating diverse sequences exhibit semantically one-to-many relationships between source and target sequences. |
| Approach: | They propose to separate diversification from generation using a general plug-and-play module that wraps around and guides an existing encoder-decoder model. |
| Outcome: | The proposed method shows that diversification and generation are separate steps in the same model and that the model is robust. |
P-React: Synthesizing Topic-Adaptive Reactions of Personality Traits via Mixture of Specialized LoRA Experts (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on personalized large language models focus on modeling explicit character profiles, while ignoring the underlying personality traits that truly shape behaviors and decision-making. |
| Approach: | They propose a personalized large language model (LLM) that captures implicit Big Five personality traits and integrates a Personality Specialization Loss to capture individual trait expressions. |
| Outcome: | The proposed model improves on Big Five personality traits and integrates a Personality Specialization Loss (PSL) to capture individual trait expressions. |
Dialogue Summarization with Mixture of Experts based on Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Existing studies for dialogue summarization use one model at a time or treat it as a black box. |
| Approach: | They propose an LLM-based approach with role-oriented routing and fusion generation to utilize mixture of experts for dialogue summarization. |
| Outcome: | The proposed approach produces informative and accurate dialogue summarization on widely used datasets. |
Simple or Complex? Complexity-controllable Question Generation with Soft Templates and Deep Mixture of Experts Model (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work on complex questions does not consider controlling complexity of generated questions. |
| Approach: | They propose an end-to-end neural complexity-controllable question generation model that incorporates a mixture of experts as the selector of soft templates to capture question similarity while avoiding the expensive construction of actual templates. |
| Outcome: | The proposed model is superior to state-of-the-art methods in both automatic and manual evaluations on two benchmark QA datasets. |
Tiny-Attention Adapter: Contexts Are More Important Than the Number of Parameters (2022.emnlp-main)
Copied to clipboard
| Challenge: | Adapter-tuning is a paradigm that transfers a pretrained language model to downstream tasks . Previously proposed adapters are all feed-forward neural networks . |
| Approach: | They propose to use tiny-attention attention with extremely small per-head dimensionality as adapters to modify hidden states at each position . they propose to average multiple attention heads' weights during deployment to reduce its inference computation cost. |
| Outcome: | The proposed adapter outperforms other adapter-tuning methods on the GLUE benchmark . it uses attention with extremely small per-head dimensionality to modify hidden states . |
On the Risk of Evidence Pollution for Malicious Social Text Detection in the Era of LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | Evidence-enhanced detectors are able to detect malicious social text, but they are prone to evidence pollution. |
| Approach: | They propose three defense strategies to mitigate evidence pollution by large language models by machine-generated text detection and a mixture of experts. |
| Outcome: | The proposed defense strategies could mitigate evidence pollution, but they faced limitations for practical employment. |
Exploring Domain Robust Lightweight Reward Models based on Router Mechanism (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in large language models have relied on the large reward model for fine-tuning, but the use of a single reward model across domains may not always be optimal. |
| Approach: | They propose to use router mechanisms to train small language models in a domain-specific manner . they use internal routers, external routers and router adapters to create a single reward model . |
| Outcome: | The proposed approach reduces parameter size while minimizing parameter size. |
MoQAE: Mixed-Precision Quantization for Long-Context LLM Inference via Mixture of Quantization-Aware Experts (2025.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to optimize large language models for long-context inference are inefficient and consume memory. |
| Approach: | They propose a mixed-precision quantization method via mixture of experts that inputs tokens into router chunk by chunk to reduce inference overhead. |
| Outcome: | The proposed method outperforms state-of-the-art KV cache quantization methods on multiple benchmark datasets. |
EpLSA: Synergy of Expert-prefix Mixtures and Task-Oriented Latent Space Adaptation for Diverse Generative Reasoning (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing models for diverse generative reasoning struggle to generate multiple unique and plausible results. |
| Approach: | They propose a model based on expert-prefix mixtures and task-oriented latent space adaptation for diverse generative reasoning. |
| Outcome: | The proposed model outperforms existing models on three types of generative reasoning tasks. |
Universal Simultaneous Machine Translation with Mixture-of-Experts Wait-k Policy (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for simultaneous machine translation require multiple models for different latency levels, resulting in large computational costs. |
| Approach: | They propose a universal SiMT model with Mixture-of-Experts Wait-k Policy to achieve the best translation quality under arbitrary latency with only one model. |
| Outcome: | The proposed model outperforms all the strong baselines under different latency levels including the state-of-the-art adaptive policy. |
A Mixture of h - 1 Heads is Better than h Heads (2020.acl-main)
Copied to clipboard
| Challenge: | Evidence has shown that multi-head attentive neural architectures are overparameterized. |
| Approach: | They propose a multi-head attentive neural architecture that “reallocates” attention heads to different inputs. |
| Outcome: | The proposed model outperforms baselines on machine translation and language modeling tasks. |
Event Representation Learning with Multi-Grained Contrastive Learning and Triple-Mixture of Experts (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods for learning event representations fail to capture hidden feature information between events. |
| Approach: | They propose a method that extends the random masked language model by incorporating a specialized MLM to capture different grammatical structures within events. |
| Outcome: | The proposed method outperforms baselines in hard similarity and transitive sentence similarity tasks, highlighting the superiority of the proposed method. |
Transformer Based Multi-Source Domain Adaptation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to improve machine learning performance are mixed experts and domain adversarial training. |
| Approach: | They investigate the problem of unsupervised multi-source domain adaptation . they combine predictions of multiple domain experts and combine them to induce a domain agnostic representation space . |
| Outcome: | The proposed methods improve models' performance while limiting learning time. |
Cross-Domain Label-Adaptive Stance Detection (2021.emnlp-main)
Copied to clipboard
| Challenge: | Stance detection is a task that focuses on the classification of a writer’s viewpoint towards a target. |
| Approach: | They propose an end-to-end unsupervised framework for out-of-domain prediction of unseen, user-defined labels. |
| Outcome: | The proposed framework shows that it can be used to predict unseen labels over strong baselines. |
SEKE: Specialised Experts for Keyword Extraction (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Keyword extraction involves identifying the most descriptive words in a document . supervised keyword extraction is based on the mixture of experts (MoE) technique . |
| Approach: | They propose a supervised keyword extraction approach based on the mixture of experts technique . they use a learnable routing sub-network to direct information to specialised experts . |
| Outcome: | The proposed approach is based on the mixture of experts (MoE) technique . experts attend to each token and integrate it with a bidirectional long-term memory network . |